Video communication is an established method of collaboration between remotely located participants. In its basic form, a video image of a remote environment is broadcast onto a local monitor allowing a local user to see and talk to one or more remotely located participants. More particularly, immersive virtual environments attempt to simulate the experience of a face-to-face interaction for participants who are, in fact, geographically dispersed but are participating and immersed within the virtual environment.
The immersive virtual environment creates the illusion that a plurality of participants, who are typically remote from each other, occupy the same virtual space. Essentially, the immersive virtual environment consists of a computer model of a three-dimensional (3D) space, called the virtual environment. For every participant in the virtual environment, there is a 3D model to represent that participant. The models are either pre-constructed or reconstructed in real time from video images of the participants. The models have sufficient detail that it is possible to know, by looking at renderings of them, the direction of their gaze within the virtual environment. In addition to participants, there can be other objects that can be represented by 3D models within the virtual environment.
Every participant and object has a virtual pose that is defined by their corresponding location and orientation within the virtual environment. The participants are typically able to control their poses so that they can move around in the virtual environment. In addition, all the participant and object 3D models are placed, according to their virtual poses, in a 3D model of the virtual environment to make a combined 3D model. A view (e.g., an image) of the combined 3D model is created for each participant to view on their computer monitor. A participant's view is rendered using computer graphics from the point of view of the participant's virtual pose.
Various forms of non-verbal communication exist when communicating physically face-to-face that are also transferred when communicating in immersive virtual environments. For instance, gaze illustrates the direction in which a person looks, and can be an important non-verbal cue in conversation. Specifically, a speaker addresses a listener in a two-way conversation usually by directly looking at the listener. Conversely, when a person listens to another person, they usually look at them. As such, gaze conveys who is speaking to whom, and who is paying attention. Thus, gaze is useful to observers who are trying to follow and participate in a conversation.
One of the problems associated with communication in an immersive virtual environment in conventional methods and systems is the lack of eye contact between participants. Two orders of visual communication can be defined as a measure of effectively communicating eye contact, and/or gaze. The first order represents eye contact between two participants. The second order represents eye contact between two participants that is observed by a third participant. Conventional video communication systems fail to effectively communicate eye contact or gaze by failing to represent various combinations of first and second orders of communication.
In the first order, it is assumed that a first and a second participant each can see the other in their respective field-of-view in the virtual environment. That is, each of the participants cannot look at the other observed participant directly, but, instead looks at the image of the other participant on an associated monitor. As such, first order gaze is communicated if the second participant sees on an associated computer monitor a face-on, eye contact view of the first participant exactly when the first participant on his or her monitor is looking at the image of the second participant.
In the second order, it is assumed that there are three participants, a first participant, a second participant, and a third participant. Each of the participants cannot look at the other observed participant directly, but, instead looks at the image of the other participant on an associated monitor. In the second order, the first participant can see the second participant in his or her field of view (FOV) of the virtual environment. Also, the third participant can see the first participant and the second participant in his or her FOV of the virtual environment. Second order gaze is communicated when the third participant sees on a monitor the first participant looking at the second participant whenever the first participant on an associated monitor is looking at the image of the second participant.
As an example of the failure to communicate gaze on a first order, participants interact with a local monitor for communicative purposes instead of the recording camera that is positioned to capture a video image of the viewing participant. In this case, the recording camera is placed to the side of the monitor to avoid interference with views of the monitor. The monitor shows a video image of a remote participant to the viewing participant. Interest of the viewing participant is naturally focused primarily on the monitor for communication. As such, the local user directly interacts with the images of the remote participant on the monitor.
However, since the recording camera is not physically positioned at the location of interest (e.g., the remote participant on the monitor) of the viewing participant, the remote participant will not see a face-on view of the viewing participant. The viewing participant appears to be avoiding eye contact by gazing off in another direction, when in fact, the viewing participant is actually viewing the remote participant. Moreover, the same problem exists at the monitor of the local user. The local user also views a video stream of the remote participant that is not face-on. In such a system, the first order of effectively communicating gaze is not successfully performed.
As an example of the failure to communicate gaze on a second order, some conventional systems try to capture a video image of the viewing participant from the viewpoint of the center of the monitor. For instance, one system typically aims a camera through a small hole in a large monitor. Another system places semi-transparent mirrors between the monitor and the viewing participant. An eye-contact view is obtained of the viewing participant. However, all the remote participants see the same view of the viewing participant, even though the viewing participant may be viewing a particular remote participant on the local monitor. Thus, all the remote participants get the impression that the viewing participant is making eye contact with them. Such systems fail to effectively communicate gaze on a second order.
Therefore, previous methods of video communication were unable to satisfactorily provide for first and second orders of communicating gaze between participants of a virtual environment.